TrackIQ Comparison Report

Generated at 2026-02-22T02:22:35.010703+00:00

Executive Summary

Workload types differ (inference vs training). Interpret metric winners cautiously for cross-workload comparisons. Overall winner: MiniCluster. MiniCluster won 8 of 9 comparable metrics. Largest deltas: performance_per_watt (+755.27%), throughput_samples_per_sec (+518.19%), latency_p50_ms (-100.00%). No regressions exceeded 5.0%. Consistency findings: none.

performance_per_watt: +755.27%throughput_samples_per_sec: +518.19%latency_p50_ms: -100.00%

AutoPerfPyResult A

Tool: autoperfpy 0.1.0
Platform: Intel64 Family 6 Model 141 Stepping 1, GenuineIntel (Windows 11)
Framework: pytorch 2.10.0+cpu
Workload: cpu_0_fp32_bs2 (inference)
Timestamp: 2026-02-21T02:59:10.681015

MiniClusterResult B

Tool: minicluster 0.1.0
Platform: CPU (Windows 11)
Framework: pytorch 2.10.0+cpu
Workload: distributed_training_validation (training)
Timestamp: 2026-02-21T02:58:27.987985

Metric Comparison

Metric AutoPerfPy MiniCluster Abs Delta % Delta Winner
communication_overhead_percentN/AN/AN/AN/AN/A
decode_tpt_msN/AN/AN/AN/AN/A
energy_per_step_joules7.34824.11003.2383-44.07%MiniCluster
latency_p50_ms42.62380.000042.6238-100.00%MiniCluster
latency_p95_ms54.68690.000054.6869-100.00%MiniCluster
latency_p99_ms55.75910.000055.7591-100.00%MiniCluster
memory_utilization_percent96.56670.000096.5667-100.00%MiniCluster
performance_per_watt0.45553.89563.4401+755.27%MiniCluster
power_consumption_watts29.071127.10271.9684-6.77%MiniCluster
scaling_efficiency_pctN/AN/AN/AN/AN/A
temperature_celsius38.140037.45860.6814-1.79%AutoPerfPy
throughput_samples_per_sec17.0790105.581088.5020+518.19%MiniCluster
tokens_per_secN/AN/AN/AN/AN/A
ttft_msN/AN/AN/AN/AN/A

Visual Overview

Consolidated graph views for quick comparison validation.

Top Normalized Deltas

Positive favors MiniCluster; negative favors AutoPerfPy.

Metric Family Deltas

Family-level signed summary of normalized deltas.

Winner Distribution

Metric-level outcome share across comparable metrics.

Confidence Distribution

Data-strength breakdown for metric conclusions.

Normalized Metric Deltas

Positive values indicate an advantage for MiniCluster; negative values favor AutoPerfPy.

Metric Family Direction Raw Delta % Normalized Delta % Visual Advantage
performance_per_wattperformancehigh+755.27%+755.27%
MiniCluster
throughput_samples_per_secperformancehigh+518.19%+518.19%
MiniCluster
latency_p50_mslatencylow-100.00%+100.00%
MiniCluster
latency_p95_mslatencylow-100.00%+100.00%
MiniCluster
latency_p99_mslatencylow-100.00%+100.00%
MiniCluster
energy_per_step_joulesefficiencylow-44.07%+44.07%
MiniCluster
power_consumption_wattsefficiencylow-6.77%+6.77%
MiniCluster
temperature_celsiusotherhigh-1.79%-1.79%
AutoPerfPy
memory_utilization_percentmemorycontext-100.00%+0.00%
context

Metric Family Delta Waterfall

Family-level mean of normalized metric deltas (context-only metrics excluded).

Family Metrics Normalized Delta % Visual Winner
performance2+636.73%
MiniCluster
latency3+100.00%
MiniCluster
efficiency2+25.42%
MiniCluster
other1-1.79%
AutoPerfPy

Metric Availability Confidence Matrix

Confidence is based on metric availability in both results.

Metric Family AutoPerfPy Available MiniCluster Available Direction Confidence
communication_overhead_percentcommunicationnonolownone
decode_tpt_msothernonolownone
energy_per_step_joulesefficiencyyesyeslowstrong
latency_p50_mslatencyyesyeslowstrong
latency_p95_mslatencyyesyeslowstrong
latency_p99_mslatencyyesyeslowstrong
memory_utilization_percentmemoryyesyescontextstrong
performance_per_wattperformanceyesyeshighstrong
power_consumption_wattsefficiencyyesyeslowstrong
scaling_efficiency_pctothernonohighnone
temperature_celsiusotheryesyeshighstrong
throughput_samples_per_secperformanceyesyeshighstrong
tokens_per_secothernonohighnone
ttft_msothernonolownone

Consistency Analysis

No consistency regressions detected or all-reduce step data was unavailable.

Platform Comparison